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WORKSHEETS FROM AN AUTO- INSTRUCTIONAL PROGRAM IN HIGH 
SCHOOL ALGEBRA WfERE USED TO COMPARE BATA PROCESSING METHODS. 

IN THE SCRIBE SYSTEM, WORKSHEET BATA IS TRANSFERRED FIRST 
ONTO SPECIAL SHEETS REQUIRING RECORDING OF ERRORS AND 
OMISSIONS ONLY, AND THEN ONTO IBM CARDS. THE IBM CARDS CAN BE 
USED IN A SPECIAL IBM 1620 ITEM DIFFICULTY ANALYSIS PROGRAM. 
THE SIMPLE MANUAL METHOD RESULTS IN A "CONDENSED TABULATION 
SHEET," WHICH HAS THE ADVANTAGES OF IMMEDIATE IMPLEMENTATION 
AND COMPACT DISPLAY OF LARGE AMOUNTS OF BATA, BUT THE 
DISADVANTAGE CF MANUAL ROW AND COLUMN COUNTS. THE DIRECT 
KEYPUNCH METHOD USES THE SAME LAYOUT AND CODING AS THE SCRIBE 
SYSTEM, BUT THE DATA ARE RECORDED DIRECTLY ONTO IBM CARDS 
WITHOUT THE INTERVENING SCRIBE SHEET. IN THIS COMPARISON, THE 
KEYPUNCH METHOD WAS RECOMMENDED, FOLLOWED BY THE CONDENSED 
TABULATION SHEET METHOD WHEN IBM EQUIPMENT IS NOT AVAILABLE. 
THE SCRIBE SYSTEM WAS RECOMMENDED ONLY WHEN A KEYPUNCH IS NOT 
AVAILABLE, DATA VOLUME IS LARGE, AND A TIME LAG IS NOT 
OBJECTIONABLE. A SAMPLE SCRIBE SHEET, INSTRUCTIONS FOR 
RECORDING WORKSHEET DATA ON SCRIBE SHEETS, A SAMPLE DISPLAY 
OF SCRIBE OUTPUT, SAMPLE ITEM ANALYSIS OUTPUT, AND A SAMPLE 
CONDENSED TABULATION SHEET ARE INCLUDED. (BB) 
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Three ^thods of Processing Student Response Data 
in Pi’ogramed Instruction^ 



Geiald L. Frincke and la^nrence M. Stolurow 



The guidelines for making valid and reliable decisions about stimulus 
control of behavior from data obtained while students are learning are an 
important but neglected part of educational technology. The research 
•worker concerned "with this problem^ ho'v-zever, soon becomes aware of the 
difficulties of reducing large amounts of da'ba. !Chis has led many to 
argue that the needed guidelines for making the complex decisions required 
in developing materials with effective stimulus control can be arrived at 
by rising small samples of students (e.g.. Eigen, 1964). The extremes of 
position concerning sanrple size are probably rooted in the position taken 
•with regard to the importance and range of individual differences in 
learning. This paper assumes that indi^vldual differences are an important 
source of •variance in learning scores, and that scores "vary over a -wide 
enough range to require substantial mmibers of students in order to find 
ans'wers to instructional questions about the stimulus control of behavior. 
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John Gilpin of the UICSM Mathematics Project developed the second 
and third methods described in this report and used them to accon5)lish a 
number of different analyses. Dr. ihul Jacobs, ETS, assisted in the 
arrangements made to use the SCRIBE system. The study involving SCRIBE 
.was made possible by the cooperation of Educational Testing Service, 

Princeton, Ne'w Jersey, and the funds made a'vailable to them on ETS 

Project 712, provided under a grant from the Carnegie Corporation of New York. 



a 



r The fact that large auantities of data will be required needs to be 

accepted and efficient -ways of coping id.th the prohlems that they present 
^ need to he sought. This report describes three methods of processing 

l’^ response data obtained from an auto-instructional program in 

f UICSM high school algebra in which students made constructed responses 

' m 

; on work sheets. One method uses a partially utomated system consisting 

k 

I of equipment developed for processing test data. The other approaches 

develop recording systems that facilitate data analysis and decision 
f making while minimizing the equipment requirement. This report describes 

f these methods and summarizes some preliminary data obtained from their 

i or 

^ use^a common set of data. 
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FIEST SOJUDY 



Use of the SCRIBE l^ystem in Processing Student *s Response 
to Frames of a ^thematics Program 

This report describes the use of the Biucational Testing Service *s 
SCRIBE system for processing student *s responses to frames of programed 
instructional material in UICSM high school algebra. The basic infoimation 
■was the student *s constructed response recorded on "worksheets. These are 
then recorded on a special SCRIBE ans'wer sheet. These answer sheets are 
used by the SCRIBE system to punch the da-ba they contain onto IBM cards. 
These cards then can be used to obtain graphic and statistical summaries 
from a computer. 

While there are many different ■ways in which one might "want to process 
a large number of "worksheets ob"tained from students who respond frequently 
over a period of hours^ some form of automation seems to be required. 

One place at which automation could be introduced would he in data re- 
duction, another would be in data collection. It is the former phase 
that is being studied here. The problem is one of concern to both research 
workers and school administrators. Some data relating to the time require- 
ments and the cost of the "work are summarized. 
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SCRIBE sheets . Appendix A contains an actual SCRIBE sheet as used 
in the present project. These sheets must he die-cut and printed to exact 
specifications for use on the SCRIBE scoring device. 

Scoring mask. Use of a single style of SCRIBE sheet for all of the 
programed parts has been made possible through the use of transcription 
masks prepared separately for each part to be recorded. This mask, in 
effect, renumbers the recording boxes on the SCRIBE sheet in accordance 
■with page and problem numbers on the -worksheet. The lecording of fixed 
ID infoimation is also facilitated through use of the mask. Mask con- 
struction is simple: A light piece of cardboard is first cut to the size 
of a SCRIBE sheet. Then, half-inch mde slots are cut in the cardboard 
so that the marking boxes of a SCRIBE sheet placed beneath the mask are 
exposed. The exposed marking boxes are numbered on the mask to correspond 
■with the desired "worksheet page and problem numbers. Openings can be 
made in the mask over the ID information boxes in such a manner that only 
the marking boxes needed to record ID infoimation are exposed. The masks 
thus produced for mrious worksheets, are quite similar except for the 
numbering of the marking boxes. Concei-vably, they could be mass produced 
if many different sets of data were to be recorded. Individual fabrication 
requires 30 to 50 minutes. 
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Recording data . The data originally recorded "by the students on their 
worksheets were scored and the results recorded on the SCRIBE sheet hy 
blackening the appropriate marking boxes on the sheet with an ordinary 
pencil. Quits were indicated by blackening one box; errors by blackening 
a different box; and correct responses require no marking at all. This 
pattern was used to minimize transcription from worksheet to SCRIBE sheets. 
Correct responses are generally more frequent than errors on program frames, 
consequently, much recording time was saved. Sample instructions for 
clerks recording the data on SCRIBE sheets are presented in Appendix B. 

Note that the clerks have to work with graded sheets since someone has to 
judge the correctness of each response. It is these data that are recorded 
on the cards, not the original responses of the students. 

SCRIBE Sheet Processing . 

Completed SCRIBE sheets iSust be sent to Educational Testing Service, 
Princeton, New ^Tersey for further processing. Currently, their SCRIBE 
system records ..he data from each SCRIBE sheet on 6 IBM cards. Ihch card 
is numbered and contains all ID data recorded on the SCRIBE sheet, as well 
as 6o columns of item data. The item data are coded on the cards as follows: 
A number one-punch in a column indicates that the item was omitted; a blank 
column represents a correct response. 

Use cf SCRIBE output cards . Before processing the data, the IBM cards 
bearing the information from the SCRIBE sheets must be sorted according 
to card number. This is done in only a few minutes using an electric card 
sorting machine. A visual data display can then be obtained by printing 
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the information from the six card groups obtained by soiting. Inter- 
pretation of the printed information can be facilitR,ted by preparing a 
set of item label IBM cards vhich simply list the SCRIBE sheet item 
number above each column to be printed. These cards arc made as follows: 
The first card contains the hundred’s digits of the SCRIBE sheet item 
numbers in the appropriate columns. The second and third cards contain, 
respectively, the tens and unit’s digits of nhe SCBIBE sheet item numbers 
in the proper columns. Once made, these cards can be used to label .the 
SCRIBE output data for any programed part. The cards a.'ce inserted in the 
card listing device followed by a deck of cards each bearing the appro- 
priate card number and containing the data of the desired subjects for a 
given programed part. Visual inspection of the output lists enables one 
to trace the progress of any individual throughout the program. Items 
can be identified by observing their SCRIBE sheet item numbers and then 
referring to the scoring mask and a SCRIBE sheet for indication of the 
worksheet page and item number. A sample of a graph5.c display created 
in the manner given above can be found in Appendix C . 

Item difficulty analysis . The SCRIBE output cards can be used to 

2 

obtain item difficulty data. A program has been witten for the IBM l620 
which accepts unmodified SCRIBE output cards as data, and produces a summary 
of correct responses, errors, omits, and errors plus omits for each 
worksheet item. Oulpuc may be in the form of IBM cards or a typewritten 
listing. Appendix D contains a sample output listing. Further statistical 
analyses of the data can be carried out using either the card output of the 
item difficulty analysis from the SCRIBE output cards themselves. 

2 

Item Analysis Program. Urbana, 111.; Univor. of 111., Training 
Research Laboratory, 1963. 
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EVALUATION 



Cost 

Materials. The follomng items enter into the cost of using the SCRIBE 
system for worksheet response analysis. 

1. SCRIBE sheets (l per 36 O worksheet responses) . 

2. Scoring mask (l per 36 O different items). 

3 . IBM output cards (6 cards per SCRIBE sheet used plus cards used 
in further analyses). 

Processing . Some costs are for the preparation and processing of the data. 

1, Data processing machine and operator time for sorting, listing 
and item analysis. 

2. Daba recording time on SCRIBE sheets (approximately 1 minute per 
60 responses including ID information). 

3« SCRIBE system time (lOO sheets processed per minute). 

Data Recording Time 

Data from forty-five (45) worksheets for the UICSM programed instruction 
Booklet, Part 104 , were recorded on SCRIBE sheets. These worksheets contain 
246 responses each. Q?wo hours and thiarty-four minutes were required. This 
is an average of about 72 responses perminute. Data from seventy ( 70 ) 

Part 104 worksheets were also recorded on SCRIBE sheets. Part 104 requires 
st\idents to make only 225 worksheet responses, 
quired four hours and foity-five minutes time. 



Recording these data re- 
This is an average of only 
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slightly more than 55 responses per minute or about 5 / 7 "^^^^ the rate for 
Part 104. More sampling of parts is obviously required to obtain sub- 
stantial infomation about the expected cost of recording data on 
SCRIBE sheets. However, one could probably be safe in estimating that 
recording will require about 1 minute per 60 items when student error 
rates are reasonably low as they were in Parts 104 and 104 * Further 
investigation will be needed to determine the efficiency of this method 
for programs with higher or lower overall error rates. 



3 

The lower rate is probably due to a more adverse item-to-ID 
information ratio. In the future it would be desirable to record 
the time for the ID information so that this could be determined. 
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SECOND STUDY 

Use of ’’Condensed Tabulation Sheets" in Recording 

Ano 1 nrr T^Tr^Y'l^c•l^c»o4■ *Po>'*P/^’vnno 

A simple manual method -was also tried. It resulted in a display like 
that shown in Figure 2.1 , which is called a "condensed tabulation sheet," 
(C.T.S. ). Advantages of this manual method are that it can be implemented 
immediately, without special preparations or equipment, and that it makes a 
large amount of data available for inspecition in a single compact display. 
Branches and other special features of programs are readily accommodated in 
the display in visually effective ways. Disadvantages are that rows cannot 
be readily interchanged in order to make patterns in the data more obvious 
to the eye, and that row and column counts, available mechanically when the 
data is on cards, must be obtained manually. 

The display, part of which is shown in Fig. 2.1 , is for UICSM-PIP. 

made in about four hours, plus two additional hours for 
planning and laying out the somewhat complicated column headings. The display 
is laid out on a sheet of Dietzgen 20 x 20 per inch graph paper, using a 
Koh-i-Noor No. 0 pen for the numerals and layout lines, and a Kbh-i-Noor 
No. 2 pen for the "wrong" and "omit" code marks. These pens and paper are 
available at any source of drafting supplies. A "slider" consisting of a 
narrow strip of graph paper marked off and labeled exactly like the column 
headings is used when entering the data and also for some kinds of readout. 

A straight edge and stylus is useful in making column counts. 

A C.T.S. as large as 260 cases by 83 O items has been satisfactorily 
made and used. The time required to make this large chart was about 1^0 hours. 
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TBIRD STUDY 

Use of the IKvI Keypunch as a Method of Recording 
Student Worksheet Performance 

A third •way of reducing students* "worksheet data is to punch TPM 
caids directly from the la'w data. The same layout and coding can he used 
as that dc merited in Section 1 for SCRIBE output cards, and the resulting 
cards are then identical to the SCRIBE cards. The time required 
(estimated) to punch cards directly is some'what greater than that required 
to transcribe the same data onto SCRIBE sheets and somewhat less than that 
required to prepare a .ondensed tabulation sheet. Howe'ver, whene'ver the 
response error rate rises much above the lo'w level usually found in data 
from programed materials, the relative efficiency of direct punching rises. 
For reducing item data from tests where 30-60% error rates are common, 
direct punching is far more efficient than either of the other two methods 
described in this report. However, error rates of this level are seldom 
found in self-instructional programs, and there "were no data in the 
UICSM-PIP units that met this condition. 
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The Three Studies Compared 

When an IBM keypunch is available, direct punching of cards is the most 
satisfactory method of reducing student worksheet data. When IBM equipment 
is not available, the condensed tabulation sheet method is indicated. It is 
the most efficient manual method for processing item data we have been able 
to find, (Compare, for instance, the method of Campbell and Caron, 1961), 

The SCRIBE system can be recommended only when a keypunch is not 
available (but service for subsequent processing of SCRIBE-punched cards is), 
the volume of data is large and a time lag on the order of weeks is not 
objectionable since the data must be sent in, processed, and returned. 
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APPENDIX A 
Sample SCRIBE Sheet 
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APPENDIX B 



Instructions for Recording Worksheet 
Data on SCRIBE Sheets 




16 



SCRIBE sheets are a lot like IBM ansver sheets, hut have the 
advantage of containing more information per page. They are designed so 
that the information on them can he automatically punched into IBM cards 
hy a device called SCRIBE. Your joh is to record the proper error and 
omission data from the student worksheets on the SCRIBE sheets. Each 
sheet must also contain certain identification information. The data 
which you are recording will he used to improve existing UICSM programed 
Instruction Project math materials and to study the effects of certain 
variables on performance within the program and on a later test. 

Begin hy obtaining a supply of SCRIBE sheets, a pencil, the 
worksheets to he recorded, and an error list mask designed for the ■VTork- 
sheet for the part which you are recording. Be certain to note that 
underlined parts, e.g. , lOk are not the same as non-underlined parts, 
e.g., lOh, 

Place the error list mask over the SCRIBE sheet. line up the 
horizontal lines on the mask with those on the SCRIBE sheet. 

The ID information is to he recorded first. Detemine the student *s 
sex hy looking at the name on the worksheet. Blacken in the rectangle 
labeled "1” in the second row of squares in the upper left-hand corner 
of the SCRIBE sheet if the student is a male. Blacken •^he number two 
rectangle if the student is a female. If you can*t te] - from the name, 
leave this blank on the SCRIBE sheet. Next, fill in the squares 
corresponding to the student *s school number and ID number. You will 
find this infoimation on the worksheet in the space labeled "ID No," 
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Tha first digit of* this number is the school number, the second tvo are 
the student’s ID number. Now fill in the spaces indicated by the arrows 
leading from the words "fill in" on the error list mask. These identify 
the part number of the worksheet. See if there is a "U" following the 
ID number on the worksheet. If there is, fill in the indicated space on 
the SCRIBE sheet. If there is no "U, " leave this space blank. 

You are now ready to record the errors and the omits. Look through 
the -vjorksheet until you come to the first error or omit. Note the page 
number and the problem number or letter of the problem done wrong or 
omitted. Find this page number on the mask and locate the problem number 
next to it or below it. Fill in the rectangle labeled "R" if the problem 
•VTas wrong. If the problem was omitted, fill in the rectangle labeled 
"N. " Some of the parts use "branching" programs^ so you will find that 
some students will skip over whole sections of problems. When you have 
finished recording all the omits and all errors, look over the SCRIBE 
sheet to be sure that you have made all your marks black and that you 
have completed the ID information in the upper left portion of the sheet. 
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Sample Graphic Display of SCRIBE Output 
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